Let activities heartbeat during worker shutdown by baekgyu-kim · Pull Request #2903 · temporalio/sdk-java

baekgyu-kim · 2026-06-08T14:19:50Z

What was changed

On the worker-shutdown path (heartbeatExecutor already shut down), HeartbeatContextImpl.heartbeat() now emits the heartbeat to the server before throwing ActivityWorkerShutdownException (previously: thrown without sending).
Throttled like the normal scheduled path (success → heartbeat interval, transient failure → short retry interval), reusing the last heartbeat's throttle state so a caller looping on heartbeat() cannot flood the server.
Send errors are swallowed (WARN-logged); ActivityWorkerShutdownException is always thrown, so a transient failure cannot mask the shutdown signal.
ActivityWorkerShutdownException Javadoc updated accordingly.

Why?

Before: on shutdown, heartbeat() threw immediately without contacting the server → no heartbeat during the awaitTermination grace period → server times the activity out and retries it → duplicate executions, despite the worker deliberately giving the activity time to finish.
After: the activity can keep heartbeating during the grace period (by catching the exception) → each call refreshes the server-side heartbeat deadline → no premature timeout/retry.
Design: emit-then-throw is the approach agreed in Add the ability to keep heartbeating while the worker is shutting down #2075; throttling keeps a manual heartbeat loop from flooding the server.

Checklist

Closes Add the ability to keep heartbeating while the worker is shutting down #2075
How was this tested:

New HeartbeatContextImplTest cases:
- emit-then-throw on the shutdown path
- exception still thrown when the send fails (gRPC and non-gRPC)
- repeated rapid calls throttled to a single send
- prompt retry after a failed send (short interval, not a full one)
- resend once the throttle interval elapses
- throttle carry-over from a preceding normal heartbeat — fails if the throttle-state write is removed
```
./gradlew :temporal-sdk:test --tests "io.temporal.internal.activity.HeartbeatContextImplTest"
./gradlew :temporal-sdk:spotlessJavaCheck
```
Any docs updates needed?
- The behavior is documented in the ActivityWorkerShutdownException Javadoc.

maciejdudko

Hi @baekgyu-kim, thank you for your contribution! It's great to see someone taking on these long standing issues. However, this is not the right implementation.

There should be a new worker option to enable heartbeating during shutdown, It should default to disabled, and when disabled, the behavior should be identical to existing behavior for backward compatibility purposes.

When the option is enabled, the heartbeat behavior should be identical to what happens during normal heartbeat when the worker is not shutting down. There should be no additional code path that calls sendHeartbeatRequest a different way, the existing mechanism should be used. The way to achieve that is to modify SyncActivityWorker.shutdown so that heartbeatExecutor.shutdown is only called after all outstanding activity tasks have finished executing.

If you need assistance with implementation, feel free to reach out on community Slack, either message me directly or post on #java-sdk channel.

baekgyu-kim · 2026-06-11T23:30:56Z

Hi @maciejdudko,
Thank you for the thoughtful review! I've reworked the PR as suggested.

It now adds an experimental WorkerOptions.Builder#setActivityHeartbeatDuringShutdown option. (default false, preserving the existing behavior)
When enabled, the heartbeat executor is shut down only after all outstanding activity tasks have finished executing, so heartbeats go through the existing mechanism without any separate code path.

Whenever you have a chance, I'd appreciate another look. Thanks again!

maciejdudko · 2026-06-12T14:52:00Z

    private PollerBehavior workflowTaskPollersBehavior;
    private PollerBehavior activityTaskPollersBehavior;
    private PollerBehavior nexusTaskPollersBehavior;
+    private boolean activityHeartbeatDuringShutdown;


The field should be named allowActivityHeartbeatDuringShutdown, the options getter should be named getAllowActivityHeartbeatDuringShutdown, and the builder setter should be named setAllowActivityHeartbeatDuringShutdown. Apply this change consistently throughout the PR.

maciejdudko · 2026-06-12T15:04:17Z

-              return null;
-            });
+    CompletableFuture<Void> shutdownFuture;
+    if (activityHeartbeatDuringShutdown) {


When interruptTasks is true (shutdownNow was called instead of shutdown), it should behave as if heartbeat during shutdown was disabled.

Suggested change

if (activityHeartbeatDuringShutdown) {

if (allowActivityHeartbeatDuringShutdown && !interruptTasks) {

maciejdudko · 2026-06-12T15:06:46Z

+   * io.temporal.client.ActivityWorkerShutdownException}, unless {@link
+   * WorkerOptions.Builder#setActivityHeartbeatDuringShutdown(boolean)} is enabled, in which case
+   * heartbeats keep working until the activity tasks finish executing.<br>


shutdownNow behavior stays the same, see comment in SyncActivityWorker.

Suggested change

* io.temporal.client.ActivityWorkerShutdownException}, unless {@link

* WorkerOptions.Builder#setActivityHeartbeatDuringShutdown(boolean)} is enabled, in which case

* heartbeats keep working until the activity tasks finish executing. 

* io.temporal.client.ActivityWorkerShutdownException}.

maciejdudko · 2026-06-12T15:43:27Z

+    /**
+     * If enabled, activities can keep heartbeating while the worker is shutting down. The activity
+     * heartbeat executor is closed only after all outstanding activity tasks have finished
+     * executing, so {@link io.temporal.activity.ActivityExecutionContext#heartbeat(Object)} behaves
+     * exactly as it does while the worker is running: heartbeats are throttled and sent to the
+     * server, which keeps the server from timing the activity out during the {@link
+     * WorkerFactory#awaitTermination(long, java.util.concurrent.TimeUnit)} grace period.
+     *
+     * <p>Note that with this option enabled activities are no longer notified of the worker
+     * shutdown by an {@link io.temporal.client.ActivityWorkerShutdownException} thrown from {@code
+     * heartbeat}, so they are expected to complete within the termination grace period on their
+     * own.
+     *
+     * <p>Defaults to false, meaning that after shutdown is requested, {@link
+     * io.temporal.activity.ActivityExecutionContext#heartbeat(Object)} stops sending heartbeats and
+     * throws {@link io.temporal.client.ActivityWorkerShutdownException}.
+     */
+    @Experimental
+    public Builder setActivityHeartbeatDuringShutdown(boolean activityHeartbeatDuringShutdown) {


We don't want to document implementation details.

Suggested change

/**

* If enabled, activities can keep heartbeating while the worker is shutting down. The activity

* heartbeat executor is closed only after all outstanding activity tasks have finished

* executing, so {@link io.temporal.activity.ActivityExecutionContext#heartbeat(Object)} behaves

* exactly as it does while the worker is running: heartbeats are throttled and sent to the

* server, which keeps the server from timing the activity out during the {@link

* WorkerFactory#awaitTermination(long, java.util.concurrent.TimeUnit)} grace period.

*

* Note that with this option enabled activities are no longer notified of the worker

* shutdown by an {@link io.temporal.client.ActivityWorkerShutdownException} thrown from {@code

* heartbeat}, so they are expected to complete within the termination grace period on their

* own.

*

* Defaults to false, meaning that after shutdown is requested, {@link

* io.temporal.activity.ActivityExecutionContext#heartbeat(Object)} stops sending heartbeats and

* throws {@link io.temporal.client.ActivityWorkerShutdownException}.

*/

@Experimental

public Builder setActivityHeartbeatDuringShutdown(boolean activityHeartbeatDuringShutdown) {

/**

* If true, activities can keep heartbeating during graceful worker shutdown (see {@link

* io.temporal.worker.WorkerFactory#shutdown WorkerFactory.shutdown}). Defaults to false,

* which means that after graceful shutdown is requested, calling {@link

* io.temporal.activity.ActivityExecutionContext#heartbeat ActivityExecutionContext.heartbeat}

* does not send a heartbeat and instead throws {@link

* io.temporal.client.ActivityWorkerShutdownException ActivityWorkerShutdownException}. This

* option is ignored by non-graceful shutdown (see {@link

* io.temporal.worker.WorkerFactory#shutdownNow WorkerFactory.shutdownNow}).

*

* Note that with this option enabled, activities are no longer notified of the worker

* shutdown by the {@link io.temporal.client.ActivityWorkerShutdownException

* ActivityWorkerShutdownException} exception, so they are expected to complete within the

* termination grace period on their own.

*/

@Experimental

public Builder setAllowActivityHeartbeatDuringShutdown(boolean allowActivityHeartbeatDuringShutdown) {

maciejdudko · 2026-06-12T16:05:39Z

+    WorkflowExecution execution = WorkflowClient.start(workflow::execute);
+    started.get();
+    testWorkflowRule.getTestEnvironment().shutdown();


There's a race condition here - shutdown() call can go through before activity worker receives the task, which will prevent the activity from running and the test will fail.

This feature will be easier to test using a standalone activity. It should work like this:

Test starts activity.

Test blocks on a semaphore 1 until activity starts.

Activity signals semaphore 1.

Activity blocks on semaphore 2 until shutdown is triggered.

Test calls shutdown().

Test signals semaphore 2.

Activity heartbeats then returns. (An exception will fail the activity.)

Test calls result() on activity handle to ensure it succeeded. (Failure will throw exception and fail the test.)

maciejdudko · 2026-06-12T16:09:51Z

+   * ActivityWorkerShutdownException}.
+   */
+  @Test
+  public void testHeartbeatingActivityCompletesDuringShutdown()


Also add a test case for when shutdownNow is called instead of shutdown.

baekgyu-kim requested a review from a team as a code owner June 8, 2026 14:19

baekgyu-kim mentioned this pull request Jun 8, 2026

Add the ability to keep heartbeating while the worker is shutting down #2075

Open

baekgyu-kim force-pushed the 2075 branch from 6ba8743 to a36e25a Compare June 10, 2026 12:42

maciejdudko requested changes Jun 11, 2026

View reviewed changes

baekgyu-kim force-pushed the 2075 branch from a36e25a to dc7f0cc Compare June 11, 2026 22:36

Add option to let activities heartbeat during worker shutdown

0d8852a

baekgyu-kim force-pushed the 2075 branch from dc7f0cc to 0d8852a Compare June 11, 2026 23:22

baekgyu-kim requested a review from maciejdudko June 11, 2026 23:22

maciejdudko requested changes Jun 12, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Let activities heartbeat during worker shutdown#2903

Let activities heartbeat during worker shutdown#2903
baekgyu-kim wants to merge 1 commit into
temporalio:masterfrom
baekgyu-kim:2075

baekgyu-kim commented Jun 8, 2026

Uh oh!

maciejdudko left a comment

Uh oh!

baekgyu-kim commented Jun 11, 2026

Uh oh!

maciejdudko Jun 12, 2026

Uh oh!

maciejdudko Jun 12, 2026

Uh oh!

maciejdudko Jun 12, 2026

Uh oh!

maciejdudko Jun 12, 2026

Uh oh!

maciejdudko Jun 12, 2026

Uh oh!

maciejdudko Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	if (activityHeartbeatDuringShutdown) {
	if (allowActivityHeartbeatDuringShutdown && !interruptTasks) {

Conversation

baekgyu-kim commented Jun 8, 2026

What was changed

Why?

Checklist

Uh oh!

maciejdudko left a comment

Choose a reason for hiding this comment

Uh oh!

baekgyu-kim commented Jun 11, 2026

Uh oh!

maciejdudko Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

maciejdudko Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

maciejdudko Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

maciejdudko Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

maciejdudko Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

maciejdudko Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants